Skip to content

BUG: rolling_* functions should not shrink window #7766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

seth-p
Copy link
Contributor

@seth-p seth-p commented Jul 16, 2014

Closes #7764.

@jreback
Copy link
Contributor

jreback commented Jul 17, 2014

can you add in tests for the rest of the rolling_* function? (I think make them work rather than raising, by returning all NaN). Not sure when that was changed (ideally if you can figure THAT out would be gr8, we could then reference that issue)

@jreback jreback added this to the 0.15.0 milestone Jul 17, 2014
@seth-p
Copy link
Contributor Author

seth-p commented Jul 17, 2014

Good point. rolling_min() and rolling_max() also have this problem. Am fixing those two, too..

@seth-p
Copy link
Contributor Author

seth-p commented Jul 17, 2014

It looks like this behavior was introduced as follows:
rolling_cov: 11/30/2012, so looks like from v0.10.0;
rolling_corr: 3/28/2014, so looks like from v0.14.0;
rolling_min/max: < 11/30/2012 (window = min(window, n0) is in the entire history of https://github.com/pydata/pandas/commits/master/pandas/algos.pyx), so from <= v0.10.0
Though short of actually installing and testing old versions, I can't be 100% certain...

@seth-p seth-p changed the title BUG: rolling_cov/corr should not shrink window BUG: rolling_* functions should not shrink window Jul 17, 2014
@jreback
Copy link
Contributor

jreback commented Jul 17, 2014

looks pretty good!

pls add a note in v0.15.0.txt - put under API changes, as it technically now will work correctly. I would do a small example their to illustrate the 'new' behavior.

@seth-p
Copy link
Contributor Author

seth-p commented Jul 17, 2014

I added a note to v0.15.0.txt, but afraid don't know how to "test" it. Let me know if I did something wrong...

- :func:`rolling_min`, :func:`rolling_max`, :func:`rolling_cov`, and :func:`rolling_corr`
now return objects with all ``NaN``s when ``len(arg) < min_periods <= window``
(like all other rolling functions do) rather than producing an error message.
For example, ``rolling_min(Series([1.]*4), min_periods=5, window=10)``
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

u put an ipython block (look at examples I. the docs for how to do this). u just need to show the behavior

@jreback
Copy link
Contributor

jreback commented Jul 18, 2014

You create a block something like this (obviously NOT the content). and its like an ipython session.

.. ipython:: python

   index = date_range('1/1/2000', periods=8)
   s = Series(randn(5), index=['a', 'b', 'c', 'd', 'e'])
   df = DataFrame(randn(8, 3), index=index,
                  columns=['A', 'B', 'C'])
   wp = Panel(randn(2, 5, 4), items=['Item1', 'Item2'],
              major_axis=date_range('1/1/2000', periods=5),
              minor_axis=['A', 'B', 'C', 'D'])

@seth-p
Copy link
Contributor Author

seth-p commented Jul 18, 2014

OK. Will try to give a more explicit example tomorrow or over the weekend.

BTW, is there an easy way for me to preview what the results will look like?

@creeson
Copy link

creeson commented Jul 18, 2014

Thank you very much guys! Appreciate all the work.

@jreback
Copy link
Contributor

jreback commented Jul 18, 2014

@seth-p you can build the docs see here: http://pandas.pydata.org/developers.html#contributing-to-the-documentation (it will be build in the dev docs when its merged as well).

@jreback
Copy link
Contributor

jreback commented Jul 21, 2014

@seth-p can you update the docs as discussed above...thxs

@seth-p
Copy link
Contributor Author

seth-p commented Jul 22, 2014

@jreback, I tried to follow http://pandas.pydata.org/developers.html#contributing-to-the-documentation, but am stumped by the error below. Any idea what that's about? Is there anything else I need to install besides sphinx? Apologies if I'm missing something obvious...

C:\Users\seth\Documents\github\pandas\doc>python make.py html
Running Sphinx v1.2.2
loading pickled environment... failed: [Errno 2] No such file or directory: 'C:\\Users\\seth\\Documents\\github\\pandas\\doc\\build\\doctrees\\environment.pickle'
[autosummary] generating autosummary for: 10min.rst, api.rst, basics.rst, categorical.rst, comparison_with_r.rst, comparison_with_sql.rst, computation.rst, contributing.rst, cookbook.rst, dsintro.rst, ..., r_interface.rst, release.rst, remote_data.rst, reshaping.rst, rplot.rst, sparse.rst, timeseries.rst, tutorials.rst, visualization.rst, whatsnew.rst
Exception occurred while building, starting debugger:
Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\sphinx\cmdline.py", line 253, in main
    warningiserror, tags, verbosity, parallel)
  File "C:\Python34\lib\site-packages\sphinx\application.py", line 139, in __init__
    self._init_builder(buildername)
  File "C:\Python34\lib\site-packages\sphinx\application.py", line 201, in _init_builder
    self.emit('builder-inited')
  File "C:\Python34\lib\site-packages\sphinx\application.py", line 400, in emit
    results.append(callback(self, *args))
  File "C:\Python34\lib\site-packages\sphinx\ext\autosummary\__init__.py", line 526, in process_generate_options
    base_path=app.srcdir)
  File "C:\Python34\lib\site-packages\sphinx\ext\autosummary\generate.py", line 110, in generate_autosummary_docs
    items = find_autosummary_in_files(sources)
  File "C:\Python34\lib\site-packages\sphinx\ext\autosummary\generate.py", line 227, in find_autosummary_in_files
    lines = f.read().splitlines()
  File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 794: character maps to <undefined>
> c:\python34\lib\encodings\cp1252.py(23)decode()
-> return codecs.charmap_decode(input,self.errors,decoding_table)[0]
(Pdb)

@jreback
Copy link
Contributor

jreback commented Jul 22, 2014

use sphinx 1.1.3 (something is broken in more recent version)

@seth-p
Copy link
Contributor Author

seth-p commented Jul 22, 2014

Will give it a shot, thanks.

Note that README.rst say: "Building the docs with Sphinx version 1.2 is broken. Use the latest stable version (1.2.1) or the older 1.1.3.". This suggests v1.2.2 should work...

@seth-p
Copy link
Contributor Author

seth-p commented Jul 22, 2014

I get the same error with Sphinx v1.1.3. :-(

@jreback
Copy link
Contributor

jreback commented Jul 22, 2014

maybe look thru some issues about building on docs on windows
I think it is non trivial
@jorisvandenbossche did a lot of work in this - he's away for a bit I believe

@seth-p
Copy link
Contributor Author

seth-p commented Jul 22, 2014

Looks like I have the exact same error @phaebz described on #5530 (comment). Afraid I can't make heads or tails out of the discussion there...

@jreback
Copy link
Contributor

jreback commented Jul 22, 2014

oh thought u were building on py 2.7 ? on windows
iirc that was. 3.3 - which I don't think works (problem with sphinx or something)

ok no biggie just edit and I'll take a look at your docs

@seth-p
Copy link
Contributor Author

seth-p commented Jul 22, 2014

I'm using 3.4.1 (64-bit, if that matters) on Windows. OK, will commit changes soon. Sorry for the inconvenience...

@seth-p
Copy link
Contributor Author

seth-p commented Jul 22, 2014

I checked in the updated docs. Travis build is still running... Let me know if you had something else in mind for the docs.

@jreback
Copy link
Contributor

jreback commented Jul 23, 2014

merged via dbb6185

@jreback jreback closed this Jul 23, 2014
@jreback
Copy link
Contributor

jreback commented Jul 23, 2014

thanks!

@jorisvandenbossche
Copy link
Member

yes, it is a python 3 issue what you got when building the docs. I should update the README to mention that for now it only fully works for python 2 (or better fix it on python 3)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Error Reporting Incorrect or improved errors from pandas Numeric Operations Arithmetic, Comparison, and Logical operations
Projects
None yet
Development

Successfully merging this pull request may close these issues.

rolling_cov and rolling_corr error out when len(series) < min_periods
4 participants